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Abstract 


Background: A substantial literature has established the role of the inner organizational setting on the implementation of 
evidence-based practices in community contexts, but very little of this research has been extended to the education sector, 
one of the most common settings for the delivery of mental and behavioral health services to children and adolescents. 
The current study examined the factor structure, psychometric properties, and interrelations of an adapted set of pragmatic 
organizational instruments measuring key aspects of the organizational implementation context in schools: (1) strategic 
implementation leadership, (2) strategic implementation climate, and (3) implementation citizenship behavior. 


Method: The Implementation Leadership Scale (ILS), Implementation Climate Scale (ICS), and Implementation Citizenship 
Behavior Scale (ICBS) were adapted by a research team that included the original scale authors and experts in the 
implementation of evidence-based practices in schools. These instruments were then administered to a geographically 
representative sample (n = 196) of school-based mental/behavioral health consultants to assess the reliability and structural 
validity via a series of confirmatory factor analyses. 


Results: Overall, the original factor structures for the ILS, ICS, and ICBS were confirmed in the current sample. The one 
exception was poor functioning of the Rewards subscale of the ICS, which was removed in the final ICS model. 
Correlations among the revised measures, evaluated as part of an overarching model of the organizational implementation 
context, indicated both unique and shared variance. 


Conclusions: The current analyses suggest strong applicability of the revised instruments to implementation of evidence- 
based mental and behavioral practices in the education sector. The one poorly functioning subscale (Rewards on the ICS) 
was attributed to typical educational policies that do not allow for individual financial incentives to personnel. Potential 
directions for future expansion, revision, and application of the instruments in schools are discussed. 
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Background 

A substantial body of implementation research has 
underscored the importance of organizational contexts 
in the successful adoption and sustainment of evidence- 
based practices (EBP) across various healthcare settings 
[1-5]. Findings indicate that even when high-quality 
implementation strategies——such as interactive training 
and post-training supports (e.g., observation and per- 
formance feedback)——are in place to facilitate profes- 
sional behavior change, implementation outcomes are 
highly variable [6-8]. Additional research suggests that 
characteristics of the inner organizational setting, or the 
immediate context in which implementation occurs, 
have a substantial influence on the use of evidence- 
based practices in routine service delivery [9-11]. As a 
result, most leading implementation frameworks pro- 
vide comprehensive coverage of “inner context” 
organizational factors [1, 3]. Conversely, inadequate 
attention to system influences is likely to cripple even 
the most well-resourced and thoughtful implementa- 
tion efforts, leading some to observe that “bad sys- 
tems trump good programs” [12]. However, there is a 
need for measures that capture key organizational 
context factors likely to set the stage for effective im- 
plementation. The current study examined the factor 
structure and psychometric properties of an adapted 
set of measures oriented to the inner organizational 
context for use in the education sector, an important 
service delivery context for delivery of behavioral 
health interventions and supports with considerable 
promise to positively impact public health of children 
and adolescents. 


Organizational implementation context 

A wide variety of organizational characteristics have 
been identified as relevant to implementation, ranging 
from local policy to leadership and infrastructure [13]. 
These constructs vary, however, in the extent to which 
they are proximal and specific to the successful adoption 
and sustainment of EBPs. The organizational implemen- 
tation context (OIC) reflects a subset of characteristics 
of the inner setting that are particularly relevant to the 
objective of EBP implementation. The OIC captures the 
factors within the immediate environment likely to influ- 
ence front-line professionals’ EBP use. Conceptualized 
using the Exploration, Preparation, Implementation, and 
Sustainment (EPIS) [1] framework for implementation in 
public service systems, key OIC constructs include 
strategic implementation leadership, strategic implemen- 
tation climate, and implementation citizenship behavior. 
These organizational constructs are considered focused 
or “strategic” in that they refer to specific organizational 
goals. This is in contrast to more general or “molar” ver- 
sions of the construct (e.g., global organizational climate 
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and culture, school climate) that, while important, provide 
a more comprehensive picture of the way an organization 
is functioning (e.g., general behavioral expectations at 
work, overall work stress) and are less directly linked to 
the strategic objective of EBP implementation [14]. 


Strategic implementation leadership 

Strategic implementation leadership is a subcomponent 
of general leadership that involves specific behaviors that 
support or inhibit implementation in service organiza- 
tions [15]. These include leaders being knowledgeable 
and able to articulate the importance of implementation 
and being supportive of staff, proactive in problem 
solving, and perseverant in the implementation process 
[15]. Importantly, strategic leadership exerts its strongest 
impact at an interactional level. Leaders who accomplish 
their strategic goals communicate regularly with staff, 
protect time during meetings to discuss strategic 
content, hold staff accountable, and provide ongoing 
feedback based on performance [16, 17]. In this way, 
strategic implementation leadership enhances the use of 
a number of “embedding mechanisms,” such as role 
modeling or setting clear criteria for rewards, [18] that 
communicate the importance of a strategic initiative and 
directly support staff use of new programs. Meta- 
analyses find that strategic leadership helps promote 
organizational change [19]. This finding is consistent 
with recent research on implementation strategies, 
which supports a link between enhanced implementa- 
tion leadership and an organizational climate that is 
conducive to EBP implementation [20]. 


Strategic implementation climate 

Defined as staff’s shared perception of the importance of 
EBP implementation [21], strategic implementation 
climate encompasses employee perceptions of the 
organizational supports and practices that help to define 
norms and expectations with regard to the implementa- 
tion of new EBPs. A positive implementation climate sig- 
nals what is expected, supported, and rewarded in 
relation to use of programs or practices [22]. Strategic 
implementation climate is supported by specific leader- 
ship behaviors that communicate those norms and 
expectations [15]. Similar to implementation leadership, 
strategic implementation climate reflects a subset of 
more general or molar organization climate, which is 
intended to reflect the entirety of the organizational 
setting. Existing research suggests that focused or stra- 
tegic climates (e.g., safety climate) are most related to 
specific outcomes [23]. 


Implementation citizenship behavior 
Citizenship behaviors are exhibited when employees go 
“above and beyond” their core job aspects or standard 
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“call of duty” to further the mission of the organization 
[24]. Applying the concept to the goal of EBP adoption 
and sustainment, implementation citizenship behaviors 
are those that demonstrate a commitment to EBP by 
keeping informed about the EBP being implemented and 
supporting colleagues to meet EBP standards [25]. 
Because they represent actual changes in the behaviors 
of front line service providers, implementation citizen- 
ship behaviors mediate the influence of implementation 
leadership and implementation climate on implementa- 
tion success [20]. 


OIC assessment instruments 

Instruments assessing the OIC constructs detailed above 
have been developed as part of a larger program of 
research focused on understanding and enhancing 
organizational factors that influence the implementation 
of mental and behavioral health programs and practices 
in public sector settings (e.g., community mental health, 
child welfare). Development of these measures has been 
largely consistent with the basic tenets of pragmatic 
measurement [26] in that they are low-burden (ie., 
brief), sensitive to change, actionable (ie., flowing into 
selection of implementation strategies [27]), and consist- 
ent with a larger framework or model (i.e., EPIS), among 
other criteria. 

The implementation leadership scale (ILS) [15, 28] 
was developed to capture strategic leadership behaviors 
that likely drive successful EBP implementation. The 
Implementation Climate Scale (ICS) [21] captures spe- 
cific aspects of the inner organizational climate that are 
likely supportive of EBP implementation. Research has 
shown that the ICS correlates moderately with, but is 
distinct from, a conceptually similar strategic climate 
measure and correlates weakly with molar climate in 
mental health and child welfare settings. Last, the 
Implementation Citizenship Behavior Scale (ICBS) [25] 
assesses the degree to which providers within an 
organization go above and beyond their typical job roles 
and expectations to support EBP implementation. To- 
gether, these measures capture three critical factors asso- 
ciated with the OIC hypothesized to impact successful 
EBP implementation. Each instrument is described in 
more details in the “Method” section. 

Although the strategic constructs assessed by the ILS, 
ICS, and ICBS are expected to be generalizable across 
settings, these measures are likely to require adaptation 
if they are to fit novel contexts of use (e.g., schools). 
Adaptation is often critical to improve the contextual 
appropriateness of instruments or practices [29]. Adap- 
tation of measurement tools can include changes to 
existing items, terminology, and definitions to ensure 
that they are relevant and comprehensible to end users 
[30, 31]. Subsequent to those adaptations, studies should 
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evaluate the extent to which the original factor structure 
is maintained to evaluate the validity of the tools in a 
new setting and provide information about their cross- 
setting utility. The current project was designed to con- 
duct such an evaluation, following the adaptation of the 
ILS, ICS, and ICBS to support the implementation of 
mental and behavioral health programming in the edu- 
cation sector. 


OIC assessment in the education sector 

Schools are the most common site for the delivery of 
behavioral health services to children and adolescents in 
the USA, a setting where upwards of 70-80% of service- 
connected youth receive care [32—36]. Consistent with 
the literature in the USA, we use “behavioral health” as 
an overarching term encompassing mental health and 
substance abuse services [37]. In schools, behavioral 
health includes a spectrum of services ranging from uni- 
versal prevention to selected and indicated interventions 
[38]. A diverse school-based behavioral health workforce 
including educators (e.g., teachers) and dedicated health- 
care personnel support this continuum of care [39]. 

School-based behavioral health consultants, who 
support systems and personnel to deliver evidence-based 
interventions across multiple levels of care, are fre- 
quently present in the education sector [38]. While they 
sometimes deliver direct services, these consultants 
often act as EBP champions (within school buildings) or 
intermediaries (across school buildings) to support im- 
plementation of behavioral health programs. Individuals 
functioning in this role are critical, given consistent evi- 
dence that school-based behavioral health services, while 
accessible, are unlikely to be evidence-based [39-41]. 
For instance, research suggests that, even when adopted, 
only 25-50% of school-based programs are implemented 
with acceptable fidelity, thus limiting their effects on 
student and school functioning [42]. 

In the education sector, the OIC reflects characteristics 
of inner organizational settings that impact implementa- 
tion efforts, such as administrator and teacher norms 
and behaviors. Although prior research has focused on 
individual-level factors for reasons of convenience and 
feasibility, multilevel assessment is needed to success- 
fully address implementation issues and install new pro- 
grams [43]. Existing research in education has tended to 
focus narrowly on measuring implementation outcomes 
such as fidelity [44], with minimal attention to capturing 
the organizational factors that specifically impact deliv- 
ery of EBPs. Careful assessment of the OIC in schools 
should consider multiple system levels and include 
perspectives of individual teachers and administrators, as 
well as organizational processes at the school and 
district levels [38]. 
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Educational researchers have previously proposed 
strong principal leadership as a requirement for adop- 
tion and use of SEL programs [45] and examined leader- 
ship qualities as important predictors of school climate 
and school improvement [46, 47], but no studies have 
investigated strategic implementation leadership, cli- 
mate, and citizenship. Although there are a number of 
general principal leadership measures with good psycho- 
metric properties [48], such as the Vanderbilt Assess- 
ment of Leadership in Education [49] and the Principal 
Instructional Management Rating Scale (PIMRS; [50, 
51], existing measures in the educational sector remain 
too broad to identify specific leadership behaviors that 
are directed at EBP adoption, delivery, and sustainment 
in schools because they assess a diffuse range of general 
leadership qualities (e.g., transformational leadership). 
Similarly, educational researchers have long examined 
the role of school climate——defined as people’s percep- 
tions of social norms, goals, values, interpersonal rela- 
tionships, teaching and learning practices, and 
organizational structures [52]——and its connection to 
wellbeing and positive outcomes among educators and 
students. This has led to the development and use of a 
range of broad school climate measures (e.g., [53]) that, 
while useful in assessing organizational health, do not 
capture specific barriers impeding implementation or 
strategic implementation climate. As a result, these in- 
struments lack utility regarding the selection of tailored 
implementation strategies to facilitate improved EBP 
use. Further, there is some educational research that has 
investigated broad (ie., non-strategic) citizenship behav- 
ior among educators and found it to be related to their 
participation in decision-making, sense of self-efficacy, 
and perceived status in the organization [54, 55]. How- 
ever, the concept of citizenship has not yet been specific- 
ally applied to the extent to which educators go beyond 
the “call of duty” to keep informed about EBPs and sup- 
port their fellow colleagues to deliver EBP with fidelity 
[56]). 

Despite their strong theoretical and empirical links to 
EBP implementation, none of the EPIS constructs have 
been studied systematically in schools, where successful 
implementation of universal EBPs can facilitate improve- 
ments in teacher behavior (e.g., instructional practices, 
interactions with students, use of reinforcement, etc.) 
that result in positive student outcomes [57]. As touched 
on above, in the education sector, existing measures of 
organizational processes have one or more of the follow- 
ing limitations: (1) they are most often either molar in 
nature (rather than specific to implementation) or 
intended for use with specific EBPs and _ not 
generalizable; (2) they lack an underlying theoretical 
framework; or (3) they do not translate to specific, prac- 
tical actions that strategically improve implementation. 
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Nevertheless, it is possible that the broad educational lit- 
erature has conceptualized implementation in ways that 
are not readily identifiable using contemporary imple- 
mentation science constructs. Finally, although the EPIS 
framework suggests that leadership, climate, and citizen- 
ship are inter-related aspects of the organizational con- 
text, education sector behavioral health research 
typically has explored them independently. This reflects 
a missed opportunity to understand the associations 
among these factors and the extent to which they may 
be complementary in promoting EBP use. 


Study aims 

The current study aims were twofold. First, consistent 
with the original authors’ call to examine the utility of 
the ILS, ICS, and ICBS in other applied contexts [15], 
we sought to evaluate the construct validity of adapted 
versions of these three pragmatic measures through 
large-scale administration to school-based behavioral 
health consultants. As indicated above, behavioral health 
consultants were selected given their frequently central 
role in EBP implementation. Additionally, it provided an 
opportunity to explore the inter-relationships among 
those factors to determine shared and unique variance. 
Through this, it provided an opportunity to confirm the 
relevance of the strategic implementation leadership, 
climate, and citizenship constructs in schools as well as 
assess potential utility of the measures themselves. 


Method 
Participants 
The sample included members from a_ state-wide, 
government-sponsored initiative on the west coast of the 
USA focused on the delivery of EBP to address youth 
behavioral health problems. Members were nominated 
by directors of regional special education agencies based 
on their commitment to engage in consultative efforts 
within their school systems. The regional agencies pro- 
vide coverage of all the geographical areas in the state 
and are inclusive of school systems operating in rural, 
suburban, and urban environments. Membership is 
maintained through participation in annual forums that 
focus on implementation of EBP and ongoing state-wide 
research and evaluation activities. The organization was 
established approximately 20 years ago following federal 
and state legislation calling for educators to implement 
individualized behavior intervention plans for youth and 
has since evolved to focus more broadly on dissemin- 
ation and implementation activities involving a con- 
tinuum of universal, targeted, and intensive supports 
(i.e., multi-tiered systems of support [38]). 

A total of 196 out of 212 total members participated 
(92%) and were included in analyses. Eighty percent of 
the respondents identified as female and 20% as male. 
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Most respondents (76%) identified their race/ethnicity as 
White/non-Hispanic, followed by 11% Hispanic or 
Latino, 6% Black or African American, 5% Asian, 2% 
American Indian or Alaska Native, 2% Other, and 1% 
Native Hawaiian or Pacific Islander. Four percent of 
respondents elected not to disclose race/ethnicity infor- 
mation. The most commonly held highest-degree-earned 
was a Master’s degree (77% of respondents), with the 
remaining respondents holding an Educational Specialist 
(EdS) degree (10%), PhD (4%), PsyD (3%), Bachelor’s 
(1%), and “Other” (4%). Within the Other degree 
category, most reported holding board certification as a 
behavior analyst. Two percent of respondents were 
between 20 and 29 years of age, 35% were 30-39, 30% 
were 40-49, 23% were 50-59, and 7% were 60-69. The 
average number of years in their current profession was 
15.8 (SD = 8.3 years). Due to missing data (< 5% overall), 
the number of participants included in some analyses 
was less than 196. Complete demographic information 
for participants is shown in Table 1. 


Table 1 Demographics of survey respondents 


Characteristic Number Percent 
Gender 
Male 39 19.9 
Female 155 79.1 
Ethnicity 
American Indian or Alaska Native 4 2.0 
Asian 10 5.1 
Black or African American 12 6.1 
Hispanic or Latino 22 11.2 
Native Hawaiian or Pacific Islander 2 1.0 
White/non-Hispanic 148 75.5 
Other 4 2.0 
Prefer not to disclose 7 3.6 
Highest degree earned 
BA/BS 1 0.5 
A 151 770 
Eds 19 97 
PsyD 6 3.1 
PhD 8 41 
Other 8 41 
Age 
20-29 4 2.0 
30-39 68 34.7 
40-49 58 29.6 
50-59 45 23.0 
60-69 14 7.1 
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Procedures 

Data were collected via an online survey, distributed 
through email. Prior to constructing the survey, re- 
searchers with expertise in school-based implementation 
adapted ILS, ICS, and ICBS items for the education con- 
text in collaboration with the developers of the original 
measures. Adaptations consisted of changing item word- 
ing to ensure construct equivalence for the target 
respondents (i.e., school-based practitioners [58]). An ef- 
fort was made to preserve the integrity of the original 
items and constructs while ensuring appropriateness to 
the school context [59]. Thus, all items from the original 
scales were maintained with changes only made to item 
wording, such as replacing the word “supervisor” with 
“school administrator,” “clinician” with “school personnel,” 
and “agency” with “school.” 

At the beginning of October 2015, members were sent 
an e-mail asking them to participate in an online survey 
examining their perceptions of the implementation of 
EBP to prevent or remediate student social, emotional, 
and behavioral problems. Weekly email reminders were 
sent for a period of up to 1 month to recruit as many 
respondents as possible. This University of Washington’s 
Human Subjects Institutional Review Board determined 
the study was exempt. As a result, no consent forms 
were collected, but information disclosures were 
presented to all participants prior to their completion of 
the online survey. Approval also was obtained by leader- 
ship from the participating state-wide organization. Re- 
searchers collaborated with the organization’s leadership 
surrounding the timing and administration of the web- 
based survey. 

The current study was part of a larger project examin- 
ing behavioral health consultants’ perceptions of the im- 
plementation of school-based EBPs and employed best 
practices in designing a web-based survey (e.g., visual 
ease, clear instructions, sending the survey, reminders, 
etc. [60]). The survey was divided in two sections: (a) re- 
spondents’ perceptions of organizational factors impact- 
ing uptake and use of universal school-based EBP and 
(b) respondents’ perceptions of facilitators and barriers 
to consulting with teachers to implement individualized 
EBP for youth with behavioral health problems. For this 
study, only the items from the first section were utilized. 
Given the importance of assessing respondents’ exposure 
to a school’s organizational context for the current 
assessment project, the survey included explicit instruc- 
tions for respondents to consider and report on the 
school in which they spent the most time. 


Measures 

As described above, all three of the OIC measures 
detailed below were adapted to ensure item wording was 
appropriate and relevant to the school context. 
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ILS 

The original ILS was developed to assess the degree to 
which leaders engage in specific behaviors that are sup- 
portive of EBP implementation. All ILS items are scored 
on a five-point, 0 (“not at all”) to 4 (“very great extent”) 
scale. In previous work, exploratory factor analysis (EFA) 
resulted in a 12-item scale with four subscales represent- 
ing proactive leadership, knowledgeable leadership, sup- 
portive leadership, and perseverant leadership. Subscale 
internal consistencies range from 0.95 to 0.98 [15]. Pre- 
vious confirmatory factor analysis (CFA) supported the 
fit of the measurement model to the data and indicated 
a higher-order implementation leadership factor with all 
sub-factors beneath it. The ILS has demonstrated ad- 
equate internal consistency reliability as well as conver- 
gent and discriminant validity from related scales [15]. 


ICS 

The original ICS was developed to assess the degree to 
which there is a strategic organizational climate support- 
ive of EBP implementation. Thirty-eight items were 
developed and evaluated based on the development 
process described above. All ICS items are scored on a 
five-point, 0 (“not at all”) to 4 (“very great extent”) scale. 
Previous EFAs resulted in a final factor structure of six 
ICS dimensions: focus on EBP, educational support for 
EBP, recognition for EBP, rewards for EBP, selection for 
EBP, and selection for openness, with subscale internal 
consistency ranging from .81 to .91. In the original 
development studies, ICS items were reduced from 38 to 
18, with three items falling under each factor. Confirma- 
tory factor analyses supported the factor structure and 
additional analyses provided evidence supporting the 
reliability and construct validity for the ICS [21]. 


ICBS 

The ICBS [25] was developed originally to assess the 
extent to which providers exceed typical expectations of 
their job to go above and beyond to support the imple- 
mentation of EBPs. Initially, 10 items were developed 
and evaluated based on the development process 
described above. In the original measure, supervisors 
assessed each of their providers’ implementation citizen- 
ship behavior. All ICBS items are scored on a five-point, 
O (“not at all”) to 4 (“great extent”), scale. EFAs 
supported the proposed two-factor structure of the Im- 
plementation Citizenship Behavior Scale (ICBS): helping 
others and keeping informed. Items were reduced from 
10 to 6, with three items falling under each factor. 
Internal consistencies range from .91 to .93. CFAs 
supported the factor structure and additional analyses 
provided evidence supporting the reliability and 
construct validity for the ICBS. 
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Data analytic approach 

To assess the construct validity of previously identified 
measurement models for the adapted scales, we con- 
ducted a series of confirmatory factor analyses (CFA) 
using weighted least squares means and variances 
(WLSMV) estimation with delta parameterization for 
the ordered-categorical scale items [61], as employed in 
Mplus [62]. WLSMV estimation allows for missing item 
data under the assumption that the data are missing at 
random. Model fit was assessed based on a preponder- 
ance of the evidence from the chi-square statistic, com- 
parative fit index (CFI; [63, 64]), the Tucker-Lewis index 
(TLI; [65]) and root mean square error of approximation 
(RMSEA; [66]) with values of the CFI and TLI greater 
than .95 and values of the RMSEA less than or equal to 
.05 as indicative of good model fit to the data. Based on 
Tabachnick and Fidell [67], we considered standardized 
factor loadings (3) less than .55 (i.e., “good”) to be indi- 
cative of a poorly performing item requiring further 
examination. We first tested the measurement models 
for each separate construct, as specified from previous 
analysis of these measures, and subsequently allowed 
modifications based on resulting model modification in- 
dices and theoretical justification. Finally, we assessed an 
overall OIC model with all three constructs to examine 
correlations among the factors. 


Results 

Summary statistics 

OIC scale and subscale item means, standard deviations, 
and coefficient alphas are shown in Table 2. Descriptive 
statistics for the 12-item ILS indicated that the Supportive 
subscale had the highest mean, while the Proactive sub- 
scale had the lowest mean and most dispersion. When 
examining the 18-item ICS, the Focus subscale had the 
highest mean and the Rewards subscale had the lowest 
mean and most dispersion. Measures of central tendency 
and dispersion for the 6-item ICBS revealed that the 
Helping Others subscale had a higher mean and lower 
dispersion than the Keeping Informed subscale. Skewness, 
kurtosis, and normality also were assessed by examining 
statistics and graph data of the response distributions for 
each of the measures and subscales. Inspection of these 
data indicated that all but one of the subscales had rela- 
tively normally distributed data and no significant skew- 
ness and kurtosis. The Rewards subscale from the ICS was 
associated with a significant positively skewed and lepto- 
kurtotic distribution. With regard to coefficient alphas, 
the overall ILS, ICS, and ICBS showed excellent internal 
consistency (ie., >.90) most subscales in the excellent 
range and ICS Focus, Education support, and Recognition 
subscales in the good range (ie., > .85), with only one of 
the 11 examined scales (Rewards from ICS—-—see below) 
yielding an alpha below .80. 


Lyon et al. Implementation Science (2018) 13:5 Page 7 of 14 
Table 2 Measure, subscale, and item means, standard deviations, and coefficient alphas 
Measure and subscales Mean SD Alpha 
Implementation Leadership Scale 23.99 2.15 990 
Proactive 533 3.13 944 
LS1. Administrator developed a plan to facilitate implementation of EBP 89 11 
LS2. Administrator removed obstacles to the implementation of EBP 71 02 
LS3. Administrator has established clear department standards for EBP 74 14 
Knowledgeable 571 3.46 967 
LS4. Administrator is knowledgeable about EBP 2.03 10 
LS5. Administrator is able to answer my questions about EBP 77 20 
LS6. Administrator knows what he or she is talking about when it comes to EBP 92 25 
Supportive 7.21 3.26 956 
LS7. Administrator recognizes employee efforts to successful implementation of EBP 2.29 19 
LS8. Administrator supports employee efforts to learn more about EBP 249 10 
LS9. Administrator supports employee efforts to use EBP 245 08 
Perseverant 5.96 3.33 968 
LS10. Administrator perseveres through the ups and downs of implementing EBP 2.06 16 
LS11. Administrator carries on through the challenges of implementing EBP 2.00 13 
LS12. Administrator reacts to critical EBP issues by openly addressing the problem(s) 1.92 16 
Implementation Climate Scale 34,99 4.54 925 
Focus 7.63 3.09 878 
CS1. One of this school’s main goals is to use EBPs effectively 2.43 21 
CS2. People in this school think that the implementation of EBPs is important 2.50 06 
CS3. Using EBP is a top priority in this district 2.72 18 
Education support 6.53 3.16 893 
ICS4. This school provides conferences, workshops, or seminars focusing on EBPs 2.21 19 
CS5. This school provides EBP trainings or in-services 241 09 
ICS6. This school provides EBP training materials, journals, etc. 1.90 20 
Recognition 6.18 3.32 854 
ICS7. School staff who use EBPs are seen as experts 2.32 2 
CS8. School staff who use EBPs are held in high esteem in this school 2.29 2 
CS9. School staff who use EBPs are more likely to be promoted 1.65 32 
Selection 6.39 3.44 966 
CS13. This school actively recruits staff who have previously used EBP 2.06 14 
CS14. This school actively recruits staff who have had education supporting EBP. 2.12 19 
CS15. This school actively recruits staff who value EBP 2.22 2 
Openness 7.25 3.16 916 
CS16. This school selects staff who are adaptable 247 09 
ICS17. This school selects staff who are flexible 251 10 
CS18. This school selects staff open to EBP 2.30 21 
Implementation Citizenship Behavior Scale 11.32 5.94 989 
Helping Others 5.89 2.99 945 
CBS1. School staff assist others to make sure they implement EBPs properly 1.93 01 
CBS2. School staff help teach EBP implementation procedures to new team members 1.96 08 
CBS3. School staff help others with responsibilities related to EBPs 2.00 04 
Keeping Informed 5.45 3.24 939 
CBS4. School staff keep informed of changes in EBPs 1.90 13 
ICBS5. School staff keep up with the latest news regarding EBPs 1.81 13 
CBS6. School staff keep up with school communications related to EBPs 1.84 10 
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Confirmatory factor analyses 

ILS 

We assessed the ILS as a hierarchical CFA with items 
loading on the four theorized first-order factors that, in 
turn, loaded on a_ second-order Implementation 
Leadership factor. Fit statistics for this model were 
x? (50, N=161)=126.33, p<.01, CFI=.997, TLI 
= .996, RMSEA = .097 (90% confidence interval = .076, 
119). All standardized factors loadings were signifi- 
cant (p<.05) and large (fs >.925). Examination of 
modification indices suggested that model fit could 
be improved (Ay? = 32.93) by correlating residual 
error terms between two items: “Our school administrator 
supports employee efforts to learn more about evidence- 
based practice” and “Our school administrator supports 
employee efforts to use evidence-based practice.” However, 
we rejected this modification based on the rationale that 
the residual correlation reflected a method-related artifact 
arising from the common wording of the items and not 
from an additional implementation-related source of co- 
variation. Fit for the modified model was y” (49, N = 161) = 
95.13, p< .01, CFI =.998, TLI =.997, RMSEA = .076 (90% 
confidence interval = .053, .099). The correlation between 
the two residual error terms was r = .616. 


ICS 

Hierarchical CFA for the ICS construct consisted of six 
first-order factors of three items each, and one second- 
order Implementation Climate factor. Fit for model was: 
x (129, N= 161) = 400.56, p< .01, CFI = .985, TLI = .983, 
RMSEA =.105 (90% confidence interval = .093, .116). 
Examination of standardized factor loadings for first- 
order factors indicated strong loadings for all indicators 
(all {6s > .917); however, one item, “School staff who use 
evidence-based practices are more likely to accumulate 
compensated time” exhibited a Heywood case (standard- 
ized factor loading >1.0) and, from inspection of model 
modification indices, substantial residual correlations 
with all other first-order factors (Ay’s between 105.57 
and 151.41). Deleting this one item did not substantially 
improve model fit; y7 (113, N= 161) = 334.53, p<.01, 
CFI = .987, TLI=.985, RMSEA=.101 (90% confidence 
interval = .089, .114); and resulted in a degraded Rewards 
scale (i.e., fs < .50), prompting deletion of this scale from 
the hierarchical CFA. Fit for the resulting model without 
the Rewards factor was acceptable: y” (85, N= 192) = 
243.43, p<.0l, CFI=.991, TLI=.989, RMSEA = .099 
(90% confidence interval = .084, .113). Among first-order 
factors, one item (School staff who use evidence-based 
practices are more likely to be promoted) had a standard- 
ized loading of (=.684; otherwise, all standardized 
factor loadings were strong (fs >.817). Standardized 
factor loadings for the second-order ICS factor ranged 
from {£§=.800 for Education Support to {= .937 for 
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Selection for Evidence-Based Programs. Although in- 
spection of modification indices suggested that a sub- 
stantial residual correlation existed between Focus and 
Education Support factors (Ay? = 61.50), and between 
two items on the Selection for Openness scale, “This 
school selects staff who are adaptable” and “This school 
selects staff who are flexible’ (Ay? =57.82), neither 
modification was accepted into the final model. 


ICBS 

Examination of the ICBS construct as a_ hierarchical 
CFA was not possible given that a measurement model 
with six items, two first-order factors, and one second- 
order factor is mathematically under-identified. Alterna- 
tively, we examined the ICBS as a two- first-order factor 
correlated (Helping Others and Keeping Informed), each 
with three indicators. Fit statistics for this model were: 
xv (8, N=176) = 37.78, p<.01, CFI=.999, TLI=.998, 
RMSEA =.145 (90% confidence interval =.101, .194. All 
standardized factors loadings were significant and large 
(all {6s > .897). The correlation between the two factors 
was r= .887. For comparison, we also tested a one-factor 
model consisting of all six ICBS items as indicators. 
Results for this model were: a (9, N=176) = 138.99, 
p<.01, CFI=.994, TLI=.991, RMSEA=.286 (90% 
confidence interval = .246, .329). As these models are 
not nested, a test of significant difference in model fit 
is not available; however, the observed decrement in 
fit for the one-factor model, combined with our pref- 
erence to adhere to the original measurement model 
of the ICBS, prompted us to retain the correlated two 
first-order factor model. No modifications to the 
model were warranted by modification indices. 


OIC model 

Correlations among the 11 retained first-order factors 
are shown in Table 3. The three instrument-specific 
models were combined into a final hierarchical CFA 
with correlations among ILS, ICS, and ICBS factors (see 
Fig. 1). Fit or this model was 7 (481, N = 193) = 1285.26, 
p<.01, CFI = .973, TLI = .971, RMSEA = .093 (90% confi- 
dence interval =.087, .099). One standardized factor 
loading for an item on the Perseverant Leadership Scale, 
“Our school administrator perseveres through the ups 
and downs of implementing evidence-based practice,” 
exhibited a Heywood case (/$ = 1.15); otherwise, all stan- 
dardized factor loadings for the first-order factors were 
within range, statistically significant (py < .05), and gener- 
ally large (M = .931). Loadings for the second-order fac- 
tors also were generally large, with the exception of the 
loading for Perseverant Leadership, which had /$ = .552. 
Correlations among the second-order factors were r= .845 
for ILS and ICS, r = .892 for ILS and ICBS, and r= .820 for 
ICS and ICBS. Residual variances for all factors were 
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Table 3 Inter-factor correlations 

Factor 2 3 4 S 6 7 8 9 10 11 
1. Proactive 611 685 539 627 678 673 15 649 648 692 
2. Knowledgeable 533 All 463 473 503 420 470 A443 478 
3. Supportive 215 568 623 579 569 573 529 567 
4. Perseverant 529 597 491 A468 A481 343 350 
5. Focus 805 540 158 74) 514 519 
6. Education support 648 758 781 605 591 
7. Recognition 564 817 583 623 
8. Selection for EBP 741 504 539 
9. Selection for openness 535 582 
10. Helping others 636 


11. Keeping informed 


All correlations were statistically significant at p < .05 
EBP evidence-based programs 


significant (p<.05) except for Keeping Informed and 
Selection for Openness. Inspection of modification indices 
showed that overall model fit could have been improved 
through a cross-loading of the Selection for Openness 
item, “This school selects staff who are adaptable,” on the 
Recognition subscale (Ay? = 220.96). 


Discussion 

The primary objectives of the current study were to (1) 
evaluate the factor structure of revised versions of the 
ILS, ICS, and ICBS following their tailoring to the edu- 
cation sector and (2) examine their inter-relationships. 
The revised versions of the instruments reflect the first 
application of these strategic inner setting constructs to 
the organizational implementation context of schools. In 
general, the factor structures proposed and validated in 
the child welfare and specialty mental health sectors by 
the original authors were confirmed in the current ana- 
lyses, with the one exception involving the removal of 
the Rewards subscale from the ICS (see following section 
for further discussion). Correlations among the mea- 
sures, evaluated as part of the overarching OIC model, 
indicated both unique and shared variance. In the next 
section, we discuss the implications of these findings for 
future research examining implementation leadership, 
climate, and citizenship behavior in school-based be- 
havioral health. 


Implementation leadership 

The adapted version of the ILS appeared to function well 
with the current data gathered from professionals work- 
ing in the education sector, supporting the relevance of 
the Proactive, Knowledgeable, Supportive, and Persever- 
ant constructs in schools. Of these, Supportive leader- 
ship yielded the highest ratings, consistent with 
administrations of the measure in other contexts [15]. A 


substantial literature has previously explored principal 
leadership and its relationship to school climate and 
student educational success [68], but the development of 
a measure of strategic implementation leadership in 
schools has the potential to produce more actionable 
information for promoting specific leader behaviors as 
they relate to EBP implementation. That is, the ILS out- 
lines specific leader behaviors that can be used to 
support implementation in school settings. As such, the 
school-specific ILS potentially could be used as a corner- 
stone in the application of leader-focused implementa- 
tion strategies. If used at the beginning of a school-based 
implementation effort, the ILS could identify a need for 
additional leadership training, coaching, and supports 
prior to initiating the active implementation phase [69]. 
Such supports could be based on existing principal- 
focused leadership enhancement interventions (e.g., 
Coaching for Improved Leadership; [70]), which could be 
tailored to support implementation leadership behaviors 
(e.g., helping principals learn to develop plans to facilitate 
EBP implementation). Alternatively, recently developed 
implementation-specific leadership interventions—such as 
the Leadership and Organizational Change for Implemen- 
tation (LOCI; [27, 69])—could be identified and subse- 
quently tailored for delivery in schools. 

Regardless, because many organizational theories identify 
leadership as a critical first step in system change—which 
impacts organizational climate and, subsequently, the indi- 
viduals working in a given setting [14, 15, 18]—it may be 
that particular priority should be placed on leveraging the 
ILS to support leadership change in schools. As work on 
strategic implementation leadership in the education sector 
progresses, it also will be important to examine the degree 
to which strategic leadership constructs interact with more 
general leadership characteristics—such as kindness, sup- 
port, and ethical leadership [71]-to impact EBP 
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implementation. It may be predicted that leaders who pos- 
sess generally positive leadership qualities and engage in 
strategic implementation leadership behaviors are most able 
to facilitate EBP adoption, delivery, and sustainment. 


Implementation climate 

Decades of research has evaluated the role of molar 
school climate and linked it to student engagement and 
achievement [72, 73], but the development of a measure 
of a focused implementation climate in schools reflects a 
novel departure from this tradition. Results of the 
current study indicated that Focus on EBP and Selection 
for Openness were the highest-rated constructs in the 
settings assessed, suggesting that these general processes 


OOOE 


OOOOE 


Fig. 1 Full confirmatory factor analysis model including measures of implementation leadership, climate, and citizenship 
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were more common than specific policies surrounding 
Recognition or Educational Supports for EBP. 

CFA results yielded one substantial change to the 
school version of the ICS. Specifically, results indicated 
that the Rewards subscale of the ICS was not function- 
ing adequately. As a result, the subscale was eliminated 
in the final school ICS model. Items on Rewards focus 
primarily on financial incentives and promotion for indi- 
vidual staff members who use EBP. Although this type 
of organizational support for implementation is observed 
relatively infrequently across contexts, it is particularly 
rare in the education sector [74]. School-level financial 
incentives for performance have been trialed [75], but 
individual-level tangible rewards for staff are used 
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infrequently due to resource and policy constraints. Be- 
cause its removal resulted in acceptable model fit for the 
ICS, and because it was generally seen as incompatible 
with standard practice in schools, the Rewards subscale 
was not retained in the final ICS model. Without the 
Rewards subscale, the revised ICS factor structure dem- 
onstrated a good fit to the data. Future research should 
not necessarily abandon the concept of rewards entirely, 
but seek to develop items that reflect the kinds of tan- 
gible rewards that are more likely to occur within the 
school context (e.g., the accumulation of discretionary or 
prep time). 

Given that a growing body of research has emphasized 
the role of organizational climate for the success of im- 
plementation efforts [1, 76], a school-specific version of 
the ICS may be useful to promote the adoption, fidelity, 
and sustained use of behavioral health programming in 
schools. Similar to the ILS, the subscales within the ICS 
could be used to inform data-driven decisions across 
different stages of the implementation process (Explor- 
ation, Preparation, Implementation, and Sustainment) to 
identify specific factors (e.g., processes for staff recogni- 
tion) that could be targeted via implementation strat- 
egies. Future research should examine the relationship 
between strategic implementation climate and molar 
school climate, as well as their differential influences on 
implementation outcomes in the education sector. 


Implementation citizenship 

The current study was the first to extend the concept of 
implementation citizenship behavior to implementation 
of behavioral health interventions in schools. The 
adapted version of the ICBS functioned well, with the 
Helping Others and Keeping Informed subscales both 
retained in the final model and good overall model fit. 
Results of the instrument administration indicated 
slightly higher ratings for Helping Others than for Keep- 
ing Informed. Both of these subscales have considerable 
relevance to schools, where professional learning com- 
munities (a.k.a., learning collaboratives) are a popular 
strategy to support educators in improving their practice 
through collaborative teams of learners [77, 78]. More- 
over, schools are inherently social contexts in which 
champions and key opinion leaders who go above and 
beyond the typical call of duty can have an impact on 
the attitudes and behaviors of coworkers to facilitate 
change [79, 80]. 

Although the ICBS is quite parsimonious, future 
research might consider potential expansions to the 
citizenship construct in schools to improve its ability to 
evaluate the construct more comprehensively. For 
instance, given that going “above and beyond” standard 
work obligations is the cornerstone of citizenship behav- 
ior, items assessing the extent to which staff take 
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sufficient initiative to participate in voluntary activities 
related to an implementation effort (e.g., extra meetings/ 
trainings, being observed by fellow teachers when it is 
not a requirement) or capitalize on opportunities to 
advocate for change by collectively working together to 
implement novel practices could be developed. Still, be- 
cause the ultimate goal is to maintain a set of pragmatic 
measures that are useful to implementation researchers, 
intermediaries, and champions within the system, gener- 
ation of new items or subscales for the school-specific 
ICBS should be done so sparingly. 


Interrelationships among scales 

Although leadership, climate, and citizenship have been 
explored separately, little research has evaluated the con- 
structs simultaneously. To date, none of this research 
has been conducted in schools. We evaluated an over- 
arching OIC model that included all three measures as a 
method of examining the relationships among the 
instruments. Consistent with the idea that the three 
instruments reflect components of a larger OIC, results 
suggest a combination of unique contributions and 
shared information across the scales (Table 3). This study 
did not examine an exhaustive list of OIC factors, but the 
findings provide support for the OIC as a construct reflect- 
ing a combination of inner setting characteristics that are 
most proximal to and influential on EBP implementation. 
Future research should attempt to replicate recent path 
models indicating that the impact of implementation 
leadership on front-line professional staff is mediated by 
changes in implementation climate [20]. 


Limitations/directions for future research 

Data were only gathered from embedded behavioral 
health consultants within schools in the USA. As inter- 
mediaries and local champions, these individuals tend to 
sit at the center of school-based behavioral health imple- 
mentation efforts. Still, they represent only one role in a 
school building among the multiple professionals who 
might be involved in implementation efforts [39]. Data 
gathered from multiple providers within each of the 
schools—or from multiple individuals across different 
professional roles—and aggregated to reflect the OIC fac- 
tors at the school level might have produced somewhat 
different results. Given that the structure of educational 
systems can vary internationally, the generalizability of 
the current findings and revised measures beyond the 
context of the USA also should be evaluated. Neverthe- 
less, this was an initial confirmatory study to examine 
whether the factor structure of existing OIC measures 
generalize to the school context. Furthermore, ILS, ICS, 
and ICBS scores were not directly linked to implementa- 
tion [81] or service recipient outcomes. Future research 
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may examine the degree to which the measures predict 
implementation outcomes (e.g., fidelity, reach, appropri- 
ateness). This study also relied on experts to revise the 
instruments and did not examine potential respondents’ 
perceptions of the relevance of the measure constructs 
and corresponding items for use within the educational 
sector. Data gathered from intended respondents could 
be used to inform strategic revisions to the measures to 
potentially further enhance their acceptability, appropri- 
ateness, and usability within schools. Furthermore, 
sample size was somewhat small for the overall factor 
analyses that included all three measures; however, 
sample sizes for individual-factor CFAs were within 
guidelines identified in the literature [82]. Nonetheless, 
future studies should examine the factor structure and 
psychometrics of these scales with larger samples. 
Finally, all three measures were evaluated in the same 
sample. Although this allowed for comparisons among 
the measures, it also carries a potential risk for model 
over-specification. Therefore, future research should 
confirm the models presented in independent samples. 


Conclusions 

This paper reports on an effort to adapt and validate 
three leading measures (ILS, ICS, ICBS) capturing differ- 
ent facets of the OIC as they are manifested surrounding 
the implementation of behavioral health programming 
in schools. The current study benefited from an existing 
foundation of carefully developed, reliable, valid, and 
theoretically coherent measures assessing strategic im- 
plementation climate, strategic implementation leader- 
ship, and implementation citizenship behavior. All but 
one of the subscales from the adapted measures func- 
tioned adequately. The one that functioned poorly—the 
ICS Rewards subscale—appeared to do so for predictable 
reasons related to known constraints associated with the 
school context. 

Despite success demonstrating construct validity in 
the current study, future applications or adaptations of 
the instruments may be necessary. Among these, adapta- 
tions could include expansion of the leadership, climate, 
or citizenship constructs to incorporate sub-constructs 
that are particularly relevant to schools. For instance, 
given the emphasis in schools on “response-to-interven- 
tion” and other data-driven approaches to applying 
academic interventions to student problems [83], it may 
be that the use of data to support EBP implementation 
reflects an important component of implementation 
climate in the education sector. 

Furthermore, administration of each of the measures 
to different types of school-based personnel—especially 
those for whom behavioral health programming is not a 
central part of their professional role (e.g., teachers, 
administrators)—may reflect a worthwhile future direction. 
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Teachers are the most common deliverers of universal be- 
havioral prevention programs in schools but might have 
different perspectives on the OIC from dedicated behav- 
ioral health consultants. Overall, this suite of pragmatic, 
reliable, and structurally valid measures is likely to support 
high-quality implementation research and practice in 
schools. Subsequent development of data-driven “assess- 
ment-to-action” frameworks that link the ILS, ICS, and 
ICBS to specific, multi-level implementation strategies 
[84, 85] may represent a next frontier for the implementa- 
tion behavioral health interventions in the education 
sector. 
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